Each process started by this implementation of Linda is multithreaded.
One thread carries out the user's computation, the other manages a
portion of tuple space. TCP is the underlying transport mechanism used
to move tuples between processes. Each process maintains two TCP
connections to every other process: the first allows the user thread
to communicate with the tuple space manager of the other process, the
second is used by the local tuple space manager to communicate with
the tuple space manager of the other process.

Consequences:
        1) Since the executable will be threaded, be sure to use
        appropriate compile and link flags for thread compatibility
        when building object files and libraries that will be linked
        with a Linda DMTCP executable. Unfortunately, this is an area
        where standards are lacking. Often, consulting the
        "pthread_create" man page will provide some useful
        information. Note, all user code will still execute within the
        context of a single thread, so the major concerns here are
        peripheral issues like consistent use of include files,
        library APIs and errno.  If some code modules in an executable
        are built with thread compatability enabled, and others
        without such compatability, there *may* be some
        inconsistencies.
        
        clc and flc attempt to do the right thing.

        2) An N-node run will require approximately 2N file
        descriptors per process. ntsnet itself will require ~2N file
        descriptors as well (an explicit connection to each process
        and the rsh/ssh connetion). Keep per process and system file
        descriptor limits in mind when attempting large runs.

        3) Linda performance and behavior will reflect the performance
        and behavior of the host OS's TCP sub-system.

This implementation accepts negated lengths for varying fields. To
help debug problems with inadvertent modifications of such fields,
this implementation offers a runtime option to check such fields by
calculating a CRC for the contents at the time of the out, at the time
of the field's retrieval and at the time of the completion of the
rd/in. If either of the latter two disagree with the former, a warning
is printed. The first of these two is a "source" side failure, the
second a "destination" side failure. Source side failures generally
indicate that the user has simply forgotten to leave the data
unmodified, while destination side failures (in the absence of a
corresponding source side failure) hint at a race -- the user thinks
the data is no longer in use, but the retrieval simply hasn't finished
yet. This check is somewhat costly in time, so it is disabled by
default.

When a large field of a tuple is copied, it is copied to storage
allocated from the user's address space during the processing of the
user invoked out. Freeing this space raises an issue due to the need
to avoid problems that could be caused when two threads concurrently
modify the storage heap. The tuple space manager knows when the tuple
has been in'ed and thus when the array storage can be freed, but this
thread cannot itself free the storage because of the potential for
conflict with the user thread. Instead, the tuple space manager places
the storage on a list to be freed the next time the user thread
executes a Linda operation. If the user thread never executes another
Linda operation, the storage will not be reclaimed. If this is a
problem, just make sure the out'er performs another Linda operation
after the tuple containing the large field is consumed---perhaps
in'ing an acknowledgment tuple or even a rdp of a non-existent tuple.

The tuple space manager satisfies its own need for storage from a
fixed sized storage pool. This size of this pool is set to a default
that can be overriden at run time.

This kernel allows for multiple redirection of tuples, which can make
the optimization more valuable to codes that move through phases in
which processes focus on different kinds of tuples in each phase. This
flexibility, however, is difficult to reconcile with the semantics of
inp and rdp, so tuples that might be the target of an inp or rdp are
not redirected.

Additional options may be specified at runtime. They follow all user
arguments and are separated from them by '--LindaOptions'. '-h' lists
available options. '-s <num>' sets the internal storage pool to <num>
bytes.  Options of the form '-X[01] <sw>' control redirection (aka
"rehashing") and enable detailed tracing.  0 disables and 1 enables a
given switch <sw>.  The following are off by default except where
noted.
        
Full Name       Abbr            Function
crcNegLen       crcNL           CRC check negated len fields
redirect        redir           Use redirection optimization (on by default)
traceComm       tComm           Trace low-level communication
traceOps        tOps            Linda op tracing
traceRedirect   tRedir          Trace tuple redirection

When any form of tracing is enabled, trace files are generated for
each process with the name 'trace-<id>-[uh].log', where id is a number
identifying the process in the range 0 to NumberOfProcesses-1. The 'u'
file traces activities instigated by the user, the 'm' file traces
activities of the tuple space manager. These files will appear in each
process's working directory. Example:

 userProg -a -b -c 1 2 3 --LindaOptions -s 2000000 -X0 redirect -X1 tOps

This sets the internal storage pool for each subprocess to 2000000 bytes,
turns off the redirection optimization and enables tracing of Linda
operations.
